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ABSTRACT 


The aim of this study is to understand how students accept an 
Intelligent Tutoring System (ITS) in terms of the Technology 
Acceptance Model. For this study, a web-based intelligent 
tutoring system was used by 38 university students for four 
weeks. Fifteen randomly selected students participated in two 
semi-structured focus group interviews which were 
transcribed and coded. Results showed that 64.12%, of the 
students perceived the system to be useful, 14.50% perceived 
ease of use, and 21.37%, reported an intention to use the 
instruction system. Among the most frequently reported 
features of usefulness of the ITS were the repeatability of 
course materials, effective and permanent learning, the 
flexibility in time and place of learning, and increased 
learning performance. While not many students found the 
system ‘fun,’ they still frequently reported the intention to use 
it. Student remarks on the perceived ease of use were less 
common than those on its perceived usefulness and intention 
to use. However, the lack of complaints about system 
usability could mean that they found it sufficient to mention 
that it was easy to learn and use. 
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1. INTRODUCTION 


Intelligent tutoring systems (ITSs) are computer-based 
instruction systems that employ methods of artificial 
intelligence to present students with opportunities for self- 
directed and individualized learning by giving intelligent help 
and guidance. The increasing popularity of ITSs means that 
computer-based instruction, combined with artificial 
intelligence and the design of web-based ITSs, have become 
prominent research areas [1]. The need for time and place- 
independent learning has advanced ITSs as an important 
teaching platform that allows students to learn at their own 
pace and receive adaptive feedback on their progress. 


Typically, an ITS contains three components: a domain 
model, a student model, and a teaching model. The domain 
model involves the contents of topics, exam questions, and the 
relationships among the instruction topics and questions. By 
monitoring the student, the student model stores information 
about their learning history, specifically the topics, login and 
logout times, duration of learning on the system, their answers 
to the questions, and test scores. The teaching model employs 
the information stored in the student model to provide them 


TAM also focuses on behavioral ITU the system and 
considers this as a predictor of the system’s success [7]. Saade 
[8] used TAM to examine how his web-based educational 
information system EISEL was accepted by users. One of his 
conclusions was that while PU had a significant influence on 
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with intelligent help and guidance. The features and degrees 
of intelligence can vary extensively from one ITS to another. 
These systems are generally domain independent, which 
means the teaching model can be reused in different domains 


[2]. 


Adoption of new learning systems is not a straightforward 
process. This also holds true for ITSs. Jensen and Wilson [3] 
investigated the reasons for the low level of ITS adoption 
among American schools. They also revealed the perceptions 
of teachers and school administrators and their effects on 
adopting such technologies. In that vein, while suggesting an 
enhanced TAM for web-based learning environments, Gong 
et al. [4] emphasized the importance of a user-centered and 
proactive design approach to learning systems to increase 
students’ perceptions of the system’s usefulness and ease of 
use as well as teachers’ willingness to adopt the technology. 
E-learning systems show their actual potential for providing 
effective learning only after they are introduced to and used 
by real users. These interactions shape user perceptions about 
system use and acceptance over time, which are important for 
assisting in the development of the system [5]. 


The ITS implemented in this study is a web-based computer 
instruction system that employs illustrations and videos along 
with text. It determines students’ knowledge levels via a pre- 
test. The learning content is logically divided into units, 
topics, and pages. If students can reach the predetermined 
score for a particular page, they are eligible to skip that page. 
While the teacher sets a maximum time limit to work on each 
page, the minimum time limit is set automatically by the 
system. The login time, time spent on each page, success rate 
on the exercises, and answers can be monitored by the teacher 
as well as by the user. The student model is updated after 
every finished exercise or movement between units. 


1.1 Technology Acceptance Model 

Research in many different technologies and subjects has 
shown the TAM to be a reliable system for predicting user 
behavior [6]. It employs the concepts of Perceived Usefulness 
(PU) and Perceived Ease of Use (PEOU) to determine users’ 
understanding and acceptance of any information system. 
While PU explains the extent to which users believe using the 
learning system will improve their performance, PEOU refers 
to how effortless they perceive system use to be [5]. PU was 
found to have a significant influence on users’ Intention to 
Use (ITU) the system. Additionally, it should be noted that 
computer self-efficacy has a strong direct effect on both 
PEOU and ITU [4]. 
ITU, PEOU did not. Thus, students may not be influenced by 
the ease of using an educational tool but still intend to use it 
because of its usefulness, which will make them perform 
better. However, if the tool is difficult to use, it may be 
perceived as a barrier to learning and diminish performance. 
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Al-Azawei et al. [9] explored the factors that affect 
technology acceptance and perceived satisfaction in a blended 
e-learning environment. Moreover, they emphasized the 
effectiveness of evaluating students’ learning individually and 
according to their learning styles and gender differences, 
which ITSs allow. The results showed that 31.25% of the 
students reported believing that e-learning was an effective 
solution that could potentially improve teaching and learning 
in higher education settings, and 18.75% of the students 
believed that e-learning could serve as an alternative to make 
instruction more appealing. 


Edmunds et al. [10] used TAM to explore students’ ICT- 
related experiences. This allowed them to conduct their 
investigation in not only school but also in work, social, and 
leisure contexts. Their findings indicate that students’ attitudes 
toward technology are primarily determined by usefulness and 
ease of use. 


Haddaji et al. [11] developed a web-based system to help 
instructors choose the most appropriate combination of 
personalization parameters for a particular course. This system 
was evaluated using TAM. Their results showed that 
instructors emphasized that the system has a high level of 
usefulness and ease of use and that they intend to use it in the 
future. 


The literature review showed that a number of studies have 
examined computer-based learning environments such as e- 
learning environments, blended learning environments, and 
web-based learning environments which employed TAM [9, 
11, 12]. However, no study has yet explored an ITS using 
TAM to our knowledge. Therefore, our study can fill a crucial 
gap in the literature by providing suggestions for improving 
future ITSs. 


The aim of this study is to explore how students accept this 
system in terms of the code schema developed by Yildiz [13] 
based on the TAM literature. This code schema will enable us 
to focus on certain aspects of technology acceptance process 
and elicit feedback from users in a real-life learning context. 
With the help of this feedback we aim to come up with design 
suggestions for the development of ITSs that more effectively 
meet students’ needs. Moreover, these will bring out 
characteristics of ITS which benefit student learning. Section 
3 describes the methodology of this study. Section 4 explains 
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how TAM was used to frame the users’ reactions to the ITS. 
Finally, the findings will be presented and discussed using the 
excerpts from focus group interviews. 


2. METHODOLOGY 


The ITS implemented in this study was intended for online 
teaching of a standardized information technologies module. 
38 Kastamonu University students from a number of 
departments selected the online course option over taking on- 
campus lectures. Students used the web-based ITS for four 
weeks of information technologies instruction. Choosing the 
off-campus option for the module, students logged on to the 
system to study any time and any place they wanted. This 
system allowed them to engage with a variety of learning 
content from informative texts to practice videos, solve 
problems at the end of the topics and receive intelligent 
feedback about their progress and guidance about which topic 
to proceed. The learning system can be accessed at 
http://79.123.169.199:8080/index.aspx . 


After four weeks, the students who used the Web-based ITS 
system took the same exam as the on-campus students. 
Following the exam, 15 of these students volunteered to 
participate in two groups of focus group interviews. The 
interviews were recorded using video cameras (1-h durations 
for each group). The researchers asked about students’ 
perceptions of the usefulness of the system, the convenience 
and difficulties they experienced using the system, and their 
ITU the system again. 


The methodology of the study is illustrated at Figure 1. 


After four 


38 Use ITS reek: ‘ 
38 monu weeks Achievement 
Focus Group Interviews 
(2 groups) 


Informative texts 

© Practice videos 

© End-of-topic questions 
© Intelligent feedback 

e Example applications 
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Figure 1: The methodology of the study 


For the analysis process, the transcribed transcriptions were 
coded by three authors based on the TAM code schema 
developed by Yildiz [13] (see Table 1). 


Table 1. The TAM code list (Yildiz, 2011) 


Perceived Usefulness (PU) Perceived Ease of Use (PEoU) Henaviora an Se 
(Code 1) Other perceived usefulness (Code 6) The Ease of Learning to Seoag megane 
(Code 1.1) Time and Place Flexibility Use the System y 
Reade) Equal Opponuanty (Code 7) The Ease of (Code 10) Thinking 
(Code 1.3) Interaction with diverse Using the System that The System is Fun 

cultures 

(Code 1.4) Immediate feedback (Code 8) The Ease of Acquiring (Code 11) Learning About 
(Code 1:5) Multimedia Usage the System Using Skills the Developments in the Area 
(Code 1.6) Increase in Motivation (Code 12) Recommending 
(Code 1.7) Active participation shes yatenn to oleae ues 
(Code 1.8) Supporting learning (Code 13) Thinking that the 
(Code 1.9) Economy Usage of the System will Increase 
(Code 1.10) Repeatability (Code 14) Thinking that It 
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(Code 1.11) Lifelong learning 


is a Good Idea to Use the System 


(Code 2) Increase in Learning Performance 


(Code 3) Efficient Learning Environment 


(Code 4) Effective Time Management 


(Code 5) Effective Learning Environment 


3. FINDINGS AND DISCUSSION 


An analysis of the answers of students who took the Web- 
based ITS option of the information technologies course 
showed that 64.12% of the students perceived it to be useful, 
14.50% considered it easy to use, and 21.37% reported an 
ITU the instruction system again. Eleven out of fifteen 
students frequently mentioned the usefulness of the ITS 
(Table 2). 


Table 2. The frequency and percentages of TAM codes 


Perceived Perceived Behavioral 
Participants Usefulness | Ease of Use Intention to 
(PU) (PEOU) Use (BI) 
f % f % f % 
Pl 8 | 88,88 | - - 1 11,11 
P2 3 | 60 1 20 1 20 
P3 9 | 81,81 | 1 9,09 1 9,09 
P4 3 | 50 1 16,66 2 33,33 
P5 11 | 50 3 13,63 8 36,36 
P6 4 | 66,66 | 1 16,66 1 16,66 
P7 4 57,14 | 2 28,57 1 14,28 
P8 7 77,77 | 1 11,11 1 11,11 
P9 1 33,33 | 1 33,33 1 33,33 
P10 7 77,77 | 1 11,11 1 11,11 
Pll 10 | 66,66 | 3 20 2 13,33 
P12 6 66,66 | 2 22,22 1 11,11 
P13 3 42,85 | 1 14,28 3 42,85 
P14 3 | 60 1 20 1 20 
P15 5 | 62,5 - - 3 37,5 
Total 84 | 64,12 | 19 14,50 28 21,37 


A detailed analysis of the codes shows that flexibility in time 
and place, repeatability, and creating an effective learning 
environment were the most strongly underlined themes in the 
PU category. These are followed by the codes increasing 
learning performance and providing an efficient learning 
environment. In PEOU category, the most emphasized code 
was the ease of using the system followed by learning to use 
the system. Among the codes in the behavioral ITU, ITU the 
system and belief in the prospective increase in the use of 
such systems were the most frequently mentioned. In the 
following discussion, the findings will be discussed in terms 
of the themes of PU, PEOU, and behavioral ITU. 


3.1 Perceived Usefulness (PU) 


University students older than 22 are becoming common and 
have different family and work commitments than their 
younger counterparts. As a form of distance education, ITS 
can also offer equal opportunity for such students by 
addressing their differing needs. For instance, a common 
learning challenge for the participants was feeling distracted 
in a 4-h lecture at a set time. One example was Participant 5, a 
mother: 


‘I wanted to be at home when taking this class (code 1.1). I 
logged on to the system when the kids were sleeping. It 
directly started from where I left off before. Exercises take a 
shorter time than in class, but still I learn better by myself 
(code 2), because the classes are so crowded. You cannot feel 
as motivated (code 1.6) to learn there. You have to pay 
attention to both what the teacher says and the exercises at the 
same time.’ 


Participant 1 said the following about the usefulness of the 
ITS: 


“When in the classroom, if I asked for help from my lecturer 
about how to apply my knowledge, that meant I somehow had 
a gap in my learning. But in this course, I was always 
redirected to the unlearned topic. And that way, I was pretty 
much ready when it came to the exercises (code 5). Of course 
there was the advantage of time and place. I could log in 
anytime I want, and I could decide how long I would be 
studying. Not two straight hours like in class. I get easily 
distracted in class. I could log in and study when I couldn’t 
sleep (code 1.1). So, I don’t miss the important basic topics. I 
couldn’t have learned as much in class as I did with this 
course (code 2).’ 


Participant 8 explained how his attitude toward learning 
changed for better with the course format: 


I was more relaxed, since I did not have to be somewhere at a 
set time (code 1.1 & 4) and learn something in a limited time. 
You set up your own rules. I think learning is more effective 
(code 5) if I am relaxed. It was one-to-one, directly addressing 
me. The content, videos, and exercises (code 1.5) were all 
self-explanatory. It made a difficult subject easy for me to 
understand (code 3).’ 


Participant 14 elaborated on the relaxing side of the ITS: 


‘I logged on to the system two days before the exam for 
review. I could recall most of it. I had not forgotten many 
things that I had to review right before the exam. This made 
my review time shorter (Code 3 & Code 4), and I was 
relaxed.’ 


Like many other participants, Participant 10 cited the 
flexibility of the system in allowing studying in different 
times and places, along with other features of the ITS that he 
perceived as contributing to his success: 


“The system enabled us to compensate for several things. You 
could not make it to class that day? It’s gone. But for ITS, 
there is no settled time (code 1.1.). You can’t understand 
something? You could repeat things over and over (code 
1.10). I was more successful this way, while I was bad at the 
first-term class (code 2). I think this is because I was 
motivated to continue studying (code 1.6) when I saw the 
expression ‘You have definitely learned this topic’ on screen.’ 


A similar sentiment was expressed by Participant 13: 


‘I got ambitious when I read the message “It’s quite likely that 
you have not learned this topic” on screen and tried to get it 
better next time (code 1.6).’ 
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Participant 15 emphasized the repetition that the system 
allows: 


‘It enabled complete learning because I made sure I learned 
every topic by repeating it. Consolidation makes better 
learning. It was a great advantage in terms of time (Code 4). I 
am not a diligent student, but I learned better (code 2) in a 
third of the time I would have spent in class.’ 


Participant 11 perceived the importance of feedback and how 
the ITS could support learning: 


‘Without feedback (Code 1.4) I could not have known why I 
got it wrong and to which page I should turn to get it right. 
It’s very sensible of the system to make me repeat my 
“missing” pages before the exam (Code 1.8). It was way 
better than going to class, because you cannot unlock the next 
pages without first learning the current page. It showed the 
degree of my learning and sometimes “forced” me to repeat a 
page. The biggest advantage is that you can return to a page 
from three weeks ago (Code 1.10).’ 


He, like many others, also cited the freedom that ITS gives to 
students to choose the length of their learning sessions 
compared to the seemingly long periods in on-campus classes: 


‘Distraction is a big problem for me. In class, I would 
normally listen and learn for the first 20 minutes but then get 
lost. The system is more comfortable, since I can use it for 
shorter but more frequent periods. I remember I was slightly 
afraid of this module at first, but I feel I was more successful 
at the end (Code 2).’ 


The ability of the ITS to understand their level of learning 
astonished some of the participants. Some of them had their 
own theories about how it could do this: 


‘I love the fact that I could take notes, and I got tested often in 
the system (code 1.7). It could even understand what we knew 
and what we did not know. It could even understand your 
level of understanding. I think it’s something to do with the 
time you spent on the page.’ 


Similarly, Participant 2 appreciated the guidance that the ITS 
provided: 


‘It even showed us what we have not learned and what we 
should study next. I liked the immediate feedback. (Code 1.4) 
In class, I would be lectured and told what I should do. Then I 
had to try and do all things together. It was not as effective. 
The course made it possible for me to go one step at a time 
(Code 5): Learn and apply it immediately.’ 


Participant 6 suggested that using the same system for other 
application-based courses would be beneficial, since it would 
enable them to participate actively: 


‘Our first-aid lessons lack one-to-one application of 
knowledge, yet that’s so important. It would be much more 
useful if we used this kind of intelligent system with visual 
exercises that tell us what we are missing (Code 1.5).’ 


Participant 13 further elaborated on the multimedia usage and 
how it resembled a classroom to him: 


‘The style of the system, the text, the narrative in the videos 
(Code 1.5) created a classroom atmosphere, but in a more 
relaxed mood.’ 


Participant 12 evaluated the effectiveness of the system by 
comparing it to the on-campus version of the same course that 
her friends had taken: 


International Journal of Computer Applications (0975 — 8887) 
Volume 182 — No. 22, October 2018 


‘It improved our skills in mathematical operations. I 
compared myself to friends who had taken the on-campus 
version. I tried to remind them of some bits of knowledge, but 
they found it difficult, some had already forgotten it 
completely (Code 5). I think this was the first time I felt this 
confident about a computer-related course.’ 


Among the codes of PU, interaction with diverse cultures 
(Code 1.3) and economy (1.9) were never mentioned by the 
participants. Although these features were not implicitly 
addressed in the design of the ITS, the reason economy was 
not a factor may be because on-campus education is free at 
the university, so students might have overlooked the fact that 
the ITS was also offered free. 


3.2 Perceived Ease of Use (PEOU) 

The codes of PEOU were less frequent than the codes of PU. 
The participants referred to PEOU briefly when speaking 
about other things. Participant 4 argued that she could have 
used this system easily when she was in high school. 
Participant 11 seconded this sentiment: 


“A secondary school student can easily use this system (Code 
7). I became quickly familiarized with it (Code 6). Anybody 
who uses a computer can do it as well. I’ve had an assumption 
since primary school that I cannot learn if I am not taught by 
the teacher. Now I see that if I am guided by feedback and 
supported by different materials (Code 8), I can learn by 
myself too. When I graduate, I don’t want to be left with the 
textbook. This system is very good because it will be easily 
accessible for years. (Code 1.11) 


During four weeks of the course, students did not contact the 
authors about usability problems. While this could mean that 
they had no issues using the system, they were asked to 
explain further. Participant 3 expressed how the option to 
‘switch’ to the on-campus lecture if he needed to reassured 
him: 


‘One of the reasons why I preferred to take the module with 
this system is that if I had difficulty using the system or 
learning with the system, I was permitted to go to class as 
well. But I never needed this (Code 6 & Code 7).’ 


Participant 9 said the following: 


‘There was the Help, which I did not recognize for weeks. I 
don’t think I needed it (Code 8). It was like prefaces of books, 
generally left unread.’ 


Similarly, Participant 7 found the ITS sufficient for her 
learning purposes. However, she had a different learning 
experience with the ITS in terms of time spent. Unlike other 
participants, who said that ITS made them learn faster, she 
purposely spent more time with the system to achieve deeper 
learning: 


‘I think I spent more time, but I prepared my own detailed 
notes. I read my friend’s notes from class too and realized that 
the ITS had extra information, especially practical information 
(Code 3). So it did not harm me to take longer on the ITS. I 
was thinking about asking the teacher if I had questions, but it 
never came to that (Code 6).’ 


Overall, all students concurred with the sentiments of 
Participant 6 by nodding with approval: 


‘It is easy to use (Code 7). A computer-literate person can use 
it easily (Code 7). Everything is easy to find with the left 
frame to choose the instruction topic to review from it (Code 
8). I would definitely recommend it to my other classmates.’ 
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3.3 Behavioral ITU 


While the participants often expressed their intentions to 
continue using ITS for learning in other courses, they took a 
critical approach that does not allow technological 
determinism. A teacher candidate, Participant 4, cited some 
limits of the system for her own use: 


‘If other courses were offered next term, I would choose to 
take them (Code 14). But for example, there are some 
modules in which we make presentations. This helps us to 
speak better and act comfortably in class, so this system 
would not be as beneficial.’ 


Participant 5, the mother, thought using ITS was a good idea 
and cautiously suggested other possible ITS-based courses to 
be offered: 


‘I said I have kids, so I chose to use this system, but I would 
still take it (Code 9) if I didn’t. It can be used for the standard 
courses that every program has in its curriculum if the 
contents change slightly for different programs. But for 
particular courses specific to programs, we generally need a 
human lecturer for guidance.’ 


Moreover, she perceived a change of her attitude about 
distance education: 


‘The system changed my attitude toward distant education for 
the better. If another course is offered using this, I would 
surely take it! (Code 14) In fact, I asked if there would be any, 
but for now there aren’t (code 11). I also recommended this to 
friends.’ (Code 12) 


Participant 3 also suggested other subjects to be taught with 
ITS: 


‘I don’t think this system will work for every subject, 
especially if it requires student-teacher contact. But I think it 
would be useful for courses that involve different 
perspectives. These would be presented to us according to our 
preferences. I would have liked it if the Art and Aesthetics 
course was offered like this course (Code 14). Then, I would 
be able to see the artworks, sometimes in 3-D, and however 
many times I want, along with the information about them. I 
could examine all of the works separately. That would be 
more permanent learning. It would save me the confusion that 
happens just before exams.’ (Code 9 & 14) 


Participant 4 did not have previous online experience, and the 
ITS also helped him form his ideas about online education: 


‘I had thought that distant learning meant I would log onto the 
system where topics are presented in a traditional way, or I 
wouldn’t log in at all, and you (teachers) cannot monitor us. I 
would take an exam at the end, and then you would check my 
score. But this course changed my thoughts about distant 
learning.’ (Code 14) 


4. CONCLUSION 


This study explored undergraduate students’ acceptance of an 
ITS using the TAM. Because of their differing time 
commitments, students in this study used the ITS to 
accommodate their needs. Moreover, some of them realized 
that the capabilities and flexibility of the ITS changed their 
attitudes toward learning and especially toward online 
learning. It is important to note that they were critical of the 
system, which allowed them to make suggestions about future 
offerings of the ITS course. 


Data collected from focus group interviews revealed that the 
most emphasized aspects of the PU of the ITS were the 
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flexibility to use it in different times and places, the ability to 
repeat material, and an effective learning environment. While 
students found the system easy to learn and use, they also 
reported that they intended to use the system and believed that 
such systems would gain popularity in the future. These 
results show that this ITS improved learning performance by 
creating an effective learning environment. Therefore, 
familiarizing teacher candidates and teachers with such 
systems can help support their adaptation to technology. 


4.1 Limitation 

The focus group interviews essentially collected the self- 
reported perceptions of the users. It should be noted that the 
results were derived from these indirect reports of use. 
Additionally, knowing about the participants’ levels of 
computer self-efficacy would have been useful, since high- 
level users would intuitively perceive the ITS to be easy to 
use. 


4.2 Suggestions 

While this ITS was developed for undergraduate education, 
future studies can explore how secondary school and high 
school students react to ITS. Additionally, ITSs designed for 
other subjects can be evaluated with TAM. 
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